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P REFACE 

This working note contains descriptions of the methods that the 
National Institutes of Health, the National Science Foundation, and the 
Office of Naval Research use in managing fundamental research. These 
descriptions were obtained by interviewing managers and other staff 
personnel in these agencies and interpreting their responses. None of 
the descriptions have yet been returned to the agencies for their comment 
or approval. This will be done in the near future. 

This work was done as part of the effort to plan the National Institute 
of Education (NIE) . If authorized by the Congress, the NIE would conduct 
research and development in the field of education. This report is one 
of a series on the Institute. The others are: 

o National Institute of Education: Preliminary Plan for the 
Proposed Institute (R-657-HEW) 

0 National Institute of Education: Methods for Managing Practice- 
oriented Research and Development (WN-7677) 

o National Institute of Education: Methods for Managing Program- 
matic Research and Development (WN-7678) 

o National Institute of Education: Organizational and Managerial 
Alternatives (WN-7679) 

o National Institute of Education: Evaluation of Methods for 
Managing Research and Devalopment (WN-7680) 

This report only describes the methods that the selected agencies use 
In managing fundamental R&D; it does not evaluate their relative merits. 
A comparative evaluation of these methods appears in WN**7680. 

Many of the R&D managers interviewed uuring this study expressed the 
need for additional study of the methods used in managing non*military R&D 
in the federal government. The literature on this subject is slight in 
comparison with the literature concerning the management of industrial and 
military R&D. This series of reports seeks to provide a basis for research 
into improved management practices for non-military federal R&D. The prin- 
cipal purpose of these reports, however, is to enable the planners of the 
National Institute of Education to benefit from the experience of other 
federal R&D agencies in developing the NIE*s R&D management procedures. 
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I, INTRODUCTION 

PURPOSE OF THIS REPORT 

This report describes four different methods that three federal 
agencies use in managing fundamental research. 

The format of description will be to treat one agency at a time by 
(1) presenting the steps that its managers and researchers actually take 
in managing research activity, and then (2) repeating this presentation 
for the other agencies. The intention is to present the data on which 
the interpretive and evaluative statements made in WN-7679 and WN-7680 
are based. This approach was taken because agreement on what people do 
is easier to get than agreement on the effects of their actions, or what 
they should do. Agreement on the effects of their actions is lacking 
because insufficient research on the management of federal R&D has been 
done. Agreement on what R&D managers should do is difficult because it 
is ultimately a question of value. 

The description for each agency will be called a paradigm , since 
not every detail and variant in what an agency does will be described. 
Each description is meant only as a model that depicts the essential 
steps in an agency's R&D management process. 

A step is deemed essential if changing it would significantly alter 
an estimate of the basic philosophy underlying the R&D management process 
being described. By looking at the essential steps, it is easier to infer 
what the basic underlying philosophy is, and how to project it onto a new 
situation, such as education R&D. This is, in fact, a meaningful definition 
of a paradigm; that is, the projection of something which is difficult 
to describe onto reality, where its consequences are observable and hence 
describable. 

The paradigms that will be treated in this report are listed in 
Table 1 together with the agencies using them. 
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Table 1 



AGENCIES TREATED AND THE METHODS THEY USE 



TO MANAGE FUNDAMENTAL RESEARCH 



Agency 



Paradigm 



National Institutes of Health 



Dual Panel Review 



National Science Foundation 



Single Panel Review 
Mail Review 



Office of Naval Research 



11^ 



'No" Review 



METHOD OF RESEARCH 

The data used to construct the management paradigms were obtained 
by interviewing federal R&D managers. Roughly half of the people inter- 
viewed were program directors and the other half were supervisory and 
staff personnel. Exhaustive coverage of every manager in an agency was 
not attempted, but rather key personnel and those recommended by key 
personnel were approached for interview. Altogether 38 managers were 
interviewed, some on repeated occasions. In addition, some data from 
the academic literature and from agency documents were used. A list of 

the people interviewed will appear in the final version of this report. 
The paradigms are a distillation of replies made by managers 

commenting on the nature and relative importance of their various acti- 
vities. Necessarily, this approach to research is vulnerable to biases 
and sometimes produces information that is difficult to verify. Neverthe- 
less, by asking all managers similar questions, and by filtering the 
responses as objectively as experience made possible, a fair representation 
of reality is thought to be presented. This approach is within the tradi- 
tion of naturalistic observation as a method of research. 

To gain clarity of exposition, some of the auxiliary mechanisms used 
by some agencies to overcome shortcomings in their management processes 
were omitted. Thus, matching the paradigm descriptions, the agencies 
interviewed, and the paradigm evaluations in WN-7680 to conclude that one 
agency does a better Job of managing research than another agency is not 
Justified. The operations and usefulness of these auxiliary mechanisms 
are discussed in Appendix B and the evaluation report, WN-7680. 
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TYPES OF R&D ACTIVITY 

This report is limited to the management of fundamental research. 
Methods for managing two other kinds of R&D activity, practice-oriented 
R&D and programmatic R&D, appear in WN*7677 and WN-yGTS respectively. 



In simplest terms fundamental research can. be described as activ^^ 
undertaken to add to our store of knowledge about basic processes. Funda- 
mental research is conducted according to the standards of disciplined 
inquiry in that experiments must be replicatable, and postulated truths 
must follow logically from stated assumptions or demonstrated fact. The 
mechanism for imposing this discipline is that a researcher *s professional 
status is determined by the quality of his work as judged by his scientific 
peer group. 

Fundamental research is conclusion-oriented as opposed to decision- 
oriented. Thus, its performers are searching for unique propositions 
about nature that have the universality and permanence of laws. Proposi- 
tions are unique in that the scientific community eventually reaches 
agreement that a proposed proposition is either true or false within the 
context assumed for its proof. The situation is opposite in decision- 
oriented research, or in development, where the objective is to find the 
''best" solution to a practical problem at hand. Usually, there will be 
several solutions, one of which is better than the others based on local 
and temporal value judgments. 

Another distinguishing characteristic of fundamental research is that 
the directions pursued are chosen primarily by the researchers themselves. 
The research community is the appropriate judge of these directions since 
progress is guided more by consideration of where solutions are possible, 
and where intuition hints that new results are likely to bel found than 
the itranedicate practical value of results. This noes not mean however, 
that practical problems can have no influence on the directions taken in 
fundamental research as a whole. Entirely new lines of research can be 
started in response to a practical need, and levels of effort among areas 
of inquiry can be adjusted to reflect relevance to practice. 
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A deeper description of the fundamental research mechanism has 
been proposed by T, S, Kuhn. The model above fails to convey adequately 
the compartmentalized nature of fundamental research, the coherence of 
results within these compartments, and the dynamics of knowledge develop- 
ment. A summary of Kuhn*s work is included as Appendix A of this report. 

TYPES OF MANAGEMENT ACTIVITY 

To facilitate presentation, the research management process will be 
broken into three types of activity: 

o Program Planning, 

o Program Development, 

o Proj^ram Evaluation. 
These categories are deliberately chosen to group together qualitatively 
similar management activities. 

Program Planning management activity is defined to include all the 
actions taken to fos ter> detect, and..lncubate research in new and ill- 
formed^axeas of injquiry. Also included are the procedures for deciding 
which new research areas will be added to the set of ongoing streams of 
activity. One example of such a new area is NIH's effort to understand 
^how to increase a person conscious control over his body's autonomic 
responses as a means of countering disease and malfunction. This is a 
fundamentally new concept in NIH, one which differs greatly from more 
traditional surgical and chemical techniques researched in NIHt The 
mechanisms of control are very imperfectly understood at the present time, 
and the concept is not clearly superior to existing prevention and cure 
modalities, but its potential explanatory power and practical utility 
seem very great. These characteristics are typical of new research areas 
in fundamental research. 

Program Development is defined to be the activity of managing the 
continuous process of refining and elaborating knowledge in an area. As 
a management process. Program Development is typically, though not always, 

^Using terminology developed in Appendix A, a new research area is 
more concisely defined as a new knowledge paradigm. 

ERLC 
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an iterative and aontiming sequence of stages involving: 
o assessment of research needs, 
o generation of project ideas, 
o selection of projects to support, 
o monitoring of project performance, 
o evaluation of project outcomes, and 
o utilization of results. 

In various management paradigms, these stages are managed in different 
ways. Sometimes they are done by program directors, sometimes by panels of 
scientists, and sometimes not at all. In most R&D, and especially when 
fundamental research is being done, action proceeds concurrently in 
several of the stages. This activity is called Program Development because 
from program management's perspective, fundamental research is evolutionary 
and expositional in nature: the goal of reserach Is adding fine structure 
and precision to a basic idea, and this happens through a sequence of pro- 
ject generation, project selection, and outcome events that develop over a 
long period of time. 

Program Evatua Hon Is jthe management activi t y of assessing what ha s 
been accomplished at som e point in„ time^bxJ>hgL„ researchers in a knowledge 
area, and the judgment of what ought to be done next. The vrays in which 
agencies accomplish this management activity is the third topic to be 
discussed in this paper. 

Discussion of management methods will focus on what is done at the 
program director level, and only occasionally at higher^ levels. This is 
necessary because of the decision to describe procedures that mariagers use 
in practice. Interactions at the higher levels are more political, and 
thus subject to greater variation and personality dependencies. Not much 
insight into ways of allocating a budget between mathematics research and 
physics research is gained by looking at the procedures agencies use. The 
influences surrounding the making of these decisions is the subject of 
studies in the science policy and political science literature. Attention 
here will center on what occurs at the interface between the performer and 
his immediate manager. In some agencies this latter individual is called 
a program officer, or science administrator. 
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II, DUAL PANEL REVIEW 

OVERVIEW 

The Dual Panel Review concept of reviewing research proposals by 
first one and then a second panel of scientists has evolved from a 
genesis in the Biologies Control Division of the Public Health Service. 
In the early 1900s , this Division established a panel of non-government 
scientists to provide technical advice on regulatory programs, and called 
it the Advisory Board for the Biologies Control Division. In 1930, this 
Board was renamed the National Advisory Health Council (NAHC) , and con- 
tinues today as an advisory board to the Surgeon General of the Public 
Health Service. 

NAHC began making grants for extramural research in 1918. For 
some years thereafter, NAHC retained the functions of reviewing appli- 
cationi:; for scientific merit, and deciding which grants would be paid. 
But, as the volume of grants increased, the workload had to be divided. 
The Council decided to create panels cf extramural scientists that 
recommend funding priorities to the Council for its use in deciding 
which grants would be paid. Each of these panels was called a ^'study 
section" and was managed administratively by a federally employed "execu- 
tive secretary." As the Council's extramural research program grew even 
larger, the Council also turned responsibility for determining scientific 
merit over to panels, retaining only the functions of reviewing the study 
sections* actions and deciding the final order of payment. 

A 1937 reorganization of federal health agencies transferred the 
Division of Biologies Standards and its advisory board, the NAHC, to the 
National Institutes ot Healf.h (NIH) , which had existed for seven years 
as an intramural laborator^^. The reorganization also added the National 
Cancer Institute and its advisory board, the National Advisory Cancer 
Council to NIH. 

NIH created the Research Grants Office in 1946 to run the panels set 
up by NAHC, and to administer the Office of Scientific Research and 
Development projects transferred to the Public Health Service at the end 
of World War II. Within a year the Research Grants Office was raised to 
division status and called the Division of Research Grants (DRG). As the 



Cancer Institute began extramural research, its applications were 
channeled through the DRG*s panels just as NAHC's. And like NAHC grants, 
grants from the Cancer Institute received final approval from the National 
Advisory Cancer Council. In the years following World War II, more 
institutes were added to NIH, and each one adopted this "Dual Review" 
method of program management. 

Until recently, the institute program management staff and the 
Council have had less influence on the allocation of. money to subject 
areas than DRG and its study sections. In the years just after the estab- 
lishment of DRG, executive secretaries did the research programming, ran 
planning workshops and conferences, and traveled to universities stimu- 
lating research applications in promising areas. In recent years the 
institutes have increased their influence on the allocation of health 
research money by doing more research programming. They are also using 
new instruments like R&D centers and research contracts, where DRG's 
management role is minimal. 

As the volume of grants awarded by the institutes grew, NAHC grants 
became a decreasing portion of all grants processed by ORG. Today, all 
the grants reviewed by DRG are paid by one of the divisions or institutes 
of NIH. Some of the NAHC authority has been transferred to NIH and some 
remains in the Public Health Service. 

The essential features of Dual Review are the following: 

o Support is given to individuals ^ not institutions, to work 
on a problem they have proposed. 

o Support is CQOarded for a limited period of time, but can 
be renewed through preseiTtation and approval of a new 
application* 

o Eadh proposal is subjected to two reviews^ first by a 

discipline-oriented peer panel to check on scientific merit, 
and then by a more problem-oriented council to check on 
program relevance. 

o The power to review grants for scientific quality and the 
power to program research projects reside in different 
managerial units. 



o 



The mcQOTity of managerial effort is devoted to evaluating 
proposals for determination of scientific merits rather than 
Program Planning, progress monitoring or Program Evaluation* 



SUMMARY 

General Characteristics 
Primary output: 

Mechanism of support: 

Managerial emphasis: 
Staffing plan: 

Program Planning 

Sources of new 
program ideas: 



Additions tc fundamental knowledge 
about basic processes or practical 
problems* 

Finite duration project grants are made 
to individual research scientists. 
Generally, these scientists are employees 
of a university, research institute, or 
firm. 

Evaluation of proposals receives far 
greater emphasis than any other mana-* 
gement activity. 

Program directors serve full-time in 
their positions. Most have had at 
least some experience as bench scientists 
in the past, but almost never return to 
the bench. 



Programs are collections of similar pro- 
jects derived from the list of active 
projects, rather than interlocking units 
of planned work. New programs start tiv^t 
in the scientific community, but once 
started are nurtured by managerial action. 



Mechanism for planning: 



Coordination: 



The same set of active projects Is 
divided along discipline lines by DRG, 
and Into medical problems by the 
Institutes. Around these focll, both 
DRG and the institutes rxm workshops 
to stimulate interest in topics which 
they feel are important. No quantitative 
planning methods are employed. 

Areas of interest are coordinated with 
other agencies through informal contact 
at the program director's level. Program 
priorities are influenced by the institute^ 
director's office. 



Program Development 

Sources of project ideas; 



Means of proposal 
review: 



Allocation of budget: 



Monitoring of 
performance: 



For the most part, project proposals are 
unsolicited; but occasionally, a scien- 
tist's interests in a program area are 
guided by contact with a program director. 

Every scientist's proposal is assigned a 
numeric score first by a panel of his 
peers (study section) and then re-reviewed 
by an institute advisory council. The 
first review is discipline-oriented and 
run by DRG; the second is more practice- 
oriented and run by the institute. 

The institute pays grants in the order 
determined by the study section's scores 
and approved by the Council. 

Substantive progress on projects is not 
closely monitored. 
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Evaluation of 
outcomes: 



No formal procedure. Reliance is 
placed on including scientific 
accomplishment as an item of consi- 
deration in the proposal evaluation 



process. 



/ 



Program Evaluation 



Mechanisra of 
evaluation: 



Most institutes have no formal procedures. 



One institute uses a structured 



conference to evaluate programs. 



ACTIVITIES 

While the most managerial effort is placed on proposal evaluation 
activities in Dual Review, Program Planning receives sore attention. The 
planning effort is directed at increasing the rate at which new research 
areas are developed by the scientific community. 

Program Planning 

In the Dual Review system of management, there are two sides in the 
planning process. One is the Division of Research Grants (DRG) , which 
takes a discipline-oriented view of the research effort; and the other 
consists of the ten categorial institutes which take a problem-oriented 
view. DRG has divided NIH's grant research effort into 48 different areas 
of inquiry and appointed an executive secretary and scientific panel for 
each one. Each area corresponds closely to the interest of a scientific 
discipline or sub-discipline. On the institutes' side this same research 
effort is divided differently; first by institute (heart, cancer, and 
others) and then (usually) by a health problem area. For both the institutes 
and DRG, these units of activiuy will be called programs. 

Neither side uses a direct method for planning new programs, but 
rather relies on indirect methods which encourage existing programs to 
grow in preferred directions. Thus, the two sides exert complementary, 
but indirect influences on the same collection of research activities. 
A common method of influencing program direction is to conduct workshops. 
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A typical workshop format is for a program director (or executive secretary 
in the case of DRG) to invite a half-dozen or so scientists expert in 
different aspects of a new research area plus some other scientists who 
might be enticed into the area to a two or three day meeting. The panel 
meets to assess the state of knowledge about the new area and to speculate 
on what kinds of projects would be most promising. The intent of these 
workshops is to draw scientific attention to the new area and to reach 
agreement on priorities for immadiate action. V\ord of these priorities 
travels quickly through scientific circles. An example of such a workshop 
is the session on sickle cell anemia held recently by the hemotology study 
section of DRG. Other indirect methods for Program Planning used by some 
institutes will be described in subsequent publications. 

In the Dual Review method, new programs are instituted in the organi- 
zation through a process of mutation. When the number of projects in a 
new research area grows sufficiently large in its parent study section, 
DRG divides the study section's workload in two, forming two study sections 
One of these study sections contains the new research area. Conversely, 
if the workload in a study section withers, DRG merges it with another 
study section. In this way, the workload is continually equalized across 
study sections, and the whole system "tracks" the research effort. By 
administrative statute, the DRG Director must review each study 
section's charter at least once every ten years, and decide whether to 
continue, divide, or merge it. A similar process occurs with respect to 
health problem areas in the institutes except that there is no time limit 
on the interval between reviews. 

The National Advisory Councils are used by the institutes a minor 
aid in research program planning, but not in the same way by each institute 
During Council meetings, where study section actions are re-reviewed, the 
institute director and program staff listen to comments made by Council 
members and translate them into programming emphases- :>ome directors 
solicit the opinion of Council members more deliberately by requesting 
issue papers from them or appointing them to task forces. Executive 
secretaries also observe these meetings and bring comments back to the 
study sections. 
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Program Development 

The dual aspects of Dual Review are clearly exhibited in the Program 
Development cycle* 

In the institutes, the principal parties are the program directors 
and the National Advisory Council. Their counterparts in DRG are the 
executive secretaries and the study sections. Each pair of counterparts 
perform similar functions in the Program Development process. 

In Dual Review, the great majority of research proposals are unsoli- 
cited, but a portion are responses to suggestions made during a personal 
visit by a program director. The size of this portion varies greatly with 
the institute; some emphasize the practice and some do not. On DRG's 
side, the executive secretaries travel some, but less than program 
directors, so the institutes are ^^iore influential in stimulating research 
proposals with this mechanism. 

The program directors stimulate proposals in areas of work judged 
by the institute staff to be particularly important to the institute's 
mission. In this way, the institute has some control over the directions 
that continuing programs take. 

Another means of influencing the direction of a developing program 
is to give applicants whose proposals are considered most relevant to the 
institute's mission advice in the preparation of their proposal. By 
knowing what catches the study section's attention, and the clearest ways 
to present ideas, the program director can considerably enchance the chances 
that an applicant's proposal will be accepted. 

Three times a year a great flood of proposals descends on the ORG. 
Referral officers sort through these proposals, and assign each one to 
one of the 48 study sections and one of the ten institutes. The executive 
secretary of the study section receives the proposal and assigns It to one 
or more study section members for in-depth review. With the executive 
secretary sitting as co-chairman along with a member of the panel, the 
study section meets to evaluate proposals. Each is discussed in turn, 
and at the end of discussion, each panelist votes a numerical score by 
secret ballot. The numerical average of these scores is the proposal's 
score. 

ERLC 
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The graded proposals are. then resorted and delivered to the institutes. 
Each institute receiving proposals from many study sections (and each 
study section grades proposals for several institutes) . The institutes 
then re-scale the study sections' rankings to equalize "tough" and "easy" 
panels. The program directors go over the list of proposals, looking for 
the few which seem to have received improper review, were disapproved by 
more than one study section member, cost over $100K, or have special pro- 
gram interest. At the Council meeting, each program director discusses 
the package of proposals in his area with the Council, bringing to its 
attention the particular proposals he has singled out. The Council goes 
over these proposals in some detail. The Council then votes to return the 
proposal to study section for another review, votes to pay the grants or 
re-assigns a priority score. Sometimes the Council changes a proposed 
project's budget, and then re-assigns a priority score. In addition to 
the proposals flagged by the program director, individual Council members 
may bring up any other proposals for discussion. Before the. meeting the 
institute mails to each Council member summaries of all proposals to be 
reviewed at the next meeting, so that prior preparation is possible. Over 
95% of the proposals considered at a Council meeting are never discussed, 
however, since the proposal workload is so great at each session^ (There 
are 48 study sections, but only 10 Councils). All proposals not discussed 
are passed "en bloc." 

The institutes have different ways of setting the order in whic^ 
approved grants are paid. One is to pay the proposals in order of priority 
score until the extramural budget is exhausted. Another is to order the 
proposals within each program cf the institute and pay further down the 
priority scale in important programs and less in others. In any case, 
there are more approved grants than money available to pay them by a 
factor of about two. The backlog of approved but unfunded grants is a 
principal statistic used by the extramural program in arguing for a larger 
extramural research budget. Applicants whose grants are paid receive an 
allotment of money to be spent over a period of five years or less, under 
some item restrictions on salaries and equipment. Minor rebudgating is 
approved by the program director, but otherwise research progress is not 
carefully monitored. Project outcomes are not evaluated in any formal way. 

O 
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From this description, it is evident that the largest fraction of 
management effort is applied to evaluation of proposals and not project 
planning, stimulation, monitoring, or evaluation. This emphasis is 
consistent with the policy of relying on the individual investigator 
for the bulk of project ideas, A flow diagram of the Program Develop- 
ment process appears in Figure 1. 

Program Evaluation 

Program Evaluation is not a significant part of the Dual Review 

paradigm as practiced by most of the institutes, although two of them 

have evolved some tactics for doing so. One of those methods is described 
in Appendix B. 

ORGANIZATION 

Structural Relationships 

An organization chart for the NIH showing the organization of two 
selected institutes and the Division of Research Grants is shown in Figure 
2. One of the institutes is divided by health problem area, the recent 
method; and the other by funding instrument, the traditional method. 

Staffing Relationships 

Program Directors , With few exceptions all the program directors have 
M,D,s or Ph,D,s and usually some experience as a bench scientists. There 
is a small level of migration from intramural research to program manage- 
ment, but none in the reverse direction. Program directors are selected 
by the institute's Associate Director for Extramural Research and have 
responsibility for a single program area. Virtually none serve another role 
besides their assigned responsibility. 

Advisory Council Members , At least six advisory council members are 
required by law to be authorities in health or science fields important to 
the institute's concern. Two are ex officio representatives, one required 



u ^ 

^. I 
c u 

O 01 
■H Oi 
U 

u «o 

M 

O 

>» u 

-So 



M G 
(0 

ai Oi 
> 

a 
«> « 
o 



10 u 

u « 

u u 

41 



»4 (A 

o c 

U ^ 

1 1 

o o 

>4 a 



rl5- 



I 



58 


• 




• 

3 


CO UQ 






« 

fH 
fH 




O 

u 


o* 





CD 

U V 

U G 9 

0> 0> ' 



c u o> 0 u • 

on •.Q •H u « 

_ o u u u a « 

•a u u a ■ ■ 

G « _ « 3 O 

o> .a< u 0 to • a 

u H a o 00 o 

U « O "H 9 C M 

< U U U (A •H Oi 



M 

iH 



o « 

o> u 

O G 

S 8 

V U 
IM CD 

O U 

u a 

V CD 



a 

U 
U 

•a « u • 

<H CO a H 

■S 8 H S 

o> « o o 

J3 C h h 



U o 
< U 



0 9 « 

> h 00 

fO U 00 

h « 9 

H »4 CO 



•H 

O hi 
M Oi 



I 

o o 
u 

0 Oi 

o 

•H U 
CO •H 

41 J3 
> U 



O U '0 O 

^ ^4 h U U 19 O 

(0 •H 60 0) o» 4 *a 

M 0) O >4 Oi 

0 J3 hi •H (B U4 O 

01 U CUT) 01 •H U 



a a • 

00 <d i • 

•H 0) 

c la CO IB 
o» fO oj 01 cd 

U K H fH 

•HO 01 (0 
U • JB > 
9 CO (0 U <0 

01 -H "ij <M 09 
t-l H (fl O 
(Q (Q O 

s-i s-^s 

Ki 9 Ki 10 •H 

o* « a a M 




> >« » 
p ^ .d 

h 4j 01 

• an n 

00 a (0 o 

(3 (Q u o u 

•H a o CO 
CO -H a « 

CD *a o u c 

9 hi o « 

u u o, > j: 

CO o H 
•H >» 
*a 01 Q) U 

> •H •H ♦ 
>4 O M4 U 

01 Wl •H O C) 
U Oi*0 «rn d 



O •H 4J 
U l4 "H 
01 hi 

(8 (D 
-CO 

a Q 

O U IM 
•H 

»H Ps U 
CQ UO C 
CO 0> 
O 0) •H 

an u , 

O (0 (0 ' 
Wl u 

a • 



o o» 
0) a 

0) a 
o c 

U •H 
U 

a 




CO 0» ^ 

o o « 

a u 01 
p « 

Wl >» 

a«M uQ 

^ 01 01 



00 d Wl 



u 

« J3 




« U 

•H ^ 
>» > *a 

*3 00 u 






u a ■ 

« -H 




Wl 01 0 • 

4: ^ J -H 


cuces. 


0 Co-< 

menil 
seci 


Proposals co 
several Inscl 



a o 

41 Wl Wl 

Wl «H •H 

au *a 




CD •9 

Wl fH Wl o 
01 3^ M 

^ • Q J3 

U O 4J 

Ma* 

cd o 00 CD 

9t u p m 

CD a H 0) 

41 ^ - 

Wl ' 



"3 2 

c * 

O « 



iH 

S 
d « 
o t* 

•H 

41 41 CI 

CO > " 

•O 9 

9 ua 



*H • 

CD *cj 

O Wl 

a 41 

o > 

Wl 41 

P4 CD 



4J O CD •H 
CD C CO 

S 19 a • 

cr d ^ U O 

Q) 4» ^ « S 

M « CO rs O 



•d 

d •H •d 

41 U « 

U 41 ^ 



www 

9 fH h ■ « 

a d CO 

01 o 0) u 
u O H ^ 




3 

I d • d 
e no • 

01 •H • 

^ O J3 U H 

o • <i 

. 01' " o 8 0 

oioiTacDWiada 

CO CD 0) 0) a'H o 

ca« uxauiMWi 
<i4Cu<««oa 



S Wl 

' CO 



d 

O M 

•H O Wl 01 

1-1 a o • rn o 

• a •H 9s Q. 01 

CO 01 U (0 CO O J3 

^ CO Wl •d Wl H 

• ^ en CO da 

0 *d CO X Wl o • 

01 9 u u m 

O U 01 h O *d fH 

Wl (A o» o» q o m 

a^ a ac4 CD iJ a 



01 



8^ 

a 0) 



8" 

Wl 

" 3 



1^ 

o o 

u u 
to 

'3<i. 



C0 
iJ 

U f 



C0 »4 



-16- 



o 

O M 

(M 0) 

4J U • 

to C fl 

fO 0) d 

« u a 

^ U 0) 0) 

•8 S 

Wi O H 

0) o a 

0) Wl IM 01 

M cw o a 



35 



O ^ 

CUTS « w ^ 

o 01 c0 • a 

h (0 (0 h M 0) 

O. (0 C3 O O U 

9 « >> U IM 

C U ^ ^ U O 

0 CO > 0) 0) 

•H O ,Q h « 

U fH 10 ^ 4J 

01 00 u o a 0) 

§ C C 73 0) i 0) 

OOIQ 0<H90<H 



(0 

o 
u 

U ' 
01 

u 




o » 

u 14 T! u o 

0 O 9 0) U 

01 ^ o B (d ^ 
•H ^ o a u 

(Q bOU 01 c 

C C CD 9 

O C •H 0) O 

•H • (J • 

u 9 00 Oi to 

c a *o c 00 a 

0) .H IM C 

a e CO 4J <M •H 4J 

g 0) 0) 0) (0 h 0) 

Q h h 01 U 9 Qt 

o u (d a u a 



•H U O 
A J3 <H 



00 (0 01 

C tn > 

H O O 
4-* O. 

4; p (0 

B O. (0 
o 

H O 



•O 9 
V u 

> CD 

0 U 
h G) 01 

8-° § 

o 

(Q 01 U 

3 »4 u 

O 0) 

1 a (0 



0) e 

01 • J3 

h 0) 4i 
U 

>» 9 « 

^ U U CD 

iH -H a» -H 

(0 U J3 

•H U w u 

on 

0) M B • 

Oi -2 

(0 O g H 

01 4^ 9 0) O 

h h 00 

CD U 00 (0 0) 

•HBO 4J 

3 »4 M CO 

I > Oi^ u 



CD 

01 01 

u u 

00 0 • 

O 01 « 

u u a 

Qi 01 «o 

I IM 9 



a I 

(0 

01 9 
H 

CO 01 00 

Qi«M o 

0) Wl 

0) B .d 

U H U 



hi 
o 



O 0) « 
O CO 

2 ^ « n 

g r *H « o» 



« 0» 
CO 4J T9 



B O 

•H U y 

^ M Oi hi 00 0) 

<J CO ^ |4 

o 2 01 n y o 

cn CD (/) O,^ (0 



• ti 

> tK N B ^ 



CD 



•H ^ -H 



U U >k 

CO O 00 w 

H 01 ^ ^ « 

u u u B (0 

01 CD 9 •H CO 4 

•H 01 < 0) f 

B 9 CO u 

•H cr « 4) I 

Q 0) • OA B 



§ 



o 



8) CO H 

TS « o d 

B « <A B CO 

0) 0) « H iH 

H d hi a a 



SU 4J 

_ O CO 

hi 9 ^ C 

(0 Oi 01 01 o 

01 ^ CD IM 

U IM U d 

01 O 01 H 

hi O hi 

09 U CO h 

M U O 

cd B 01 «M 

J3 H hi U 

4J hi o 01 *a 

O. 9 J3 CO 

01 01 HO) 

hi hi hi 

•H 01 • 

9 • 

cr B (0 hi fH 

' 0) H H 01 d 

CO iH -a > o 



Si 



O 9 
hi U 
Oi 

fH 

CO 

M CO 



I 



i 



0) *a iJ I i» fa IM H 

SI 8 ""^ 

* *H • CD • 

SO) 9 ^ B 01 M 00 
0) m o. o N B o 
o 0) a<H •H 0) >4 
*a>>4CQMfHaOi 



B 

O 01 o • 

•H hi W OB 

M 9 U 

CO M 4J a 

a H 01 01 

O U 00-r-) 

fH CD O 

fH B 9 hi 

< M Oi 



00 hi H 

B O • 

•H a 00 CD 

U C U 

o 9 o a 

M CD 01 

•H 00*0 

B B O 

X « U Oi 



3 



ERIC 



-17- 



33 
M 

2; 



U 



c 
o 



N 

00 
M 
O 



00 



o 

M-l 

o 

CO » 
• M 

O 

o 

Q) 
Q 



C CO 

o u 

^ s 

CO (d 

> o 

o 

•» M 

V4 cd 

O 0) 

4J CO 

U 0) 

0) P< 

'H MH 

Q O 









CO 




CO 






c 






0 




0 


•H 






u 






0 






0) 


*c 




CO 


0 












(d 






0) 






CO 


> 








W 









o 

O 

§ 



0) 



1 















0 






iH 


no 




0 


c 






(d 












>> 




CO 


OC >i 




P4 


0 00 
tH 0 


CO 


no 


0 iH 


§ 


0) 


•H 0 




*rl 


0 




iH 




0 






0 








? 

1 


T 





CO 

c 
o 

•H 

U 

(d 

(d 'H 

^ s 
i 

0) S 

a o 
o o 



to 



cu ^ 

u o 

4J (d 

•H CU 

U CO 



V4 




0 




u 




0 




0) 




V4 


iH 


•H 


cd 


P 


lur 








i 


0 




CO 




CO 




< 





00 

o 

iH 
O 
'H 
PQ 

iH 

a 

o 

iH 

I. 









0 


CO 


CO 






iH 


0) 


0) 




CO 


0 


0 


CO 




0) 


•M 


c 


(d 


0) 


CO 


Cd 


0) 


(U 


0 


(d 


B 


•H 


CO 


c 


0) 


0 




•H 


0) 


CO 


•u 


CO 


Q 


•H 


•H 


CO 








Q 




H 


iH 


CO 






(d 


(d 




iH 






c 


CO 


(d 


CO 


0 


0 


iH 




CO 




•H 


(d 


c 


•H 


iH 


4J 


'H 


0 


H 


(d 


•H 


Kl 








4J 


0) 


•H 


•M 


0) 


c 


XJ 




MM 


c 




(d 


0) 






! 


T 




T 


0 

1 



H iH (d 

O (d 4J 

w c c 

O O 0) 

'H «d M-l 

Q ^3 O 



! I 



o 
u 

o 

0) 

H iH 
Cd 

o u 

I CO w 



1^ 



CO 



CO 

c 



o 

a 







CO 






Q) 






to 




Q) 


Cd 






Q) 






CO 




u 


'H 






Q 










CO 






c 


Cd 




M 


0 Q) 






'H ^ 




iH 


00 0 


0 


Cd 


0 u 


u 


c 


iH *J 


c 


>.» 


0 CO 


0) 




U 




U 


D no 


'H 


Cd 




Q 


:2s 





o 
o 

0) 

H iH 

'H Cd 

Q ^ 

o 

o n 

i CO u 



o 
(d 

CO 
0^ 



00 

c 

.s 

cd 

H 



o 
o 

M iH 
'H Cd 

p n 
2 

o u 

CO w 
< M 



ERLC 



-18- 



from the Department of Defense, and the other from the Veterans Adminis- 
tration — and as selected, they are generally scientists. Of the re- 
maining six positions, one or two are usually lay representatives and 
the rest scientists. Advisory Council members are nominated by the 
institute director but must be approved up the line to the Secretary of 
HEV7. The term of service is four years. Council members are paid $100 
per day for their services. 

Referial Officers » The Division of Research Grants' Director selects 
referral officers from the ranks of executive secretaries to serve on a 
part-time ba^/.s. 

Executive Secretaries . In qualifications, background, and term of 
service, executive secretaries are indistinguishable from program directors. 
They are selected by the Director of the Research Grants Division in con- 
sultation with panel members and others. There is some migration from 
executive secretary positions to program director positions, but virtually 
none the other way. 

Study Section Members > The executive secretary and the study section 
chairman recommend replacements for panel members from the community of 
scientists. These nominations are then approved up the line to the Sec- 
retary of HEW, but seldom denied. The term of office is four years. The 
study section chairman of chosen by the executive secretary. Panelists 
are paid a $100 consulting fee for their services. 



III. SINGLE PANEL REVIEW 



OVERVIEW 

The National Science Foundation's Biological and Social Science 
Divisions use a reduced form of Dual Review which will be called Single 
Panel Review . It Is essentially half of the Dual Panel Review system, 
since the basic paradigm Includes only one panel and one program director 
In a partnership relationship. The Single Review program director performs 
all of the tasks performed by both the executive secretary and the program 
director In Dual Panel Review, except that he need not compose summaries 
which communicate the deliberations of one panel to the other. 

The essential features of Single Panel Review are the following: 
o Support is given to individuals^ not to institutions, to work 

on a problem they have proposed, 
o Support is csLOarded for a limited period of time, but can be 

renewed through presentation and approval of a new application, 
o Some proposals are submitted to a panel of the applicant's 

discipline peers for their comment and opinion, 
o The majority of managerial effort is devoted to selecting a 
set of projects thought to optimally balance the vajrCous lines 
of fundamental research being pursued. 

SUMMARY 

General Characteristics 

Primary output: Additions to fundamental knowledge about 

basic processes or practical problems. 

Mechanisms of support: Finite duration project grants are made 

to individual research scientists. 
Generally, these scientists are employees 
of a university, research institute, or 
firm. 
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Managerial emphasis: The majority of research effort is 

devoted to selecting the set of pro- 
jects that is thought to optimally 
balance the lines of fundamental 
research being pursued, including the 
prospects for normal science progress 
and revolutionary discovery. 

Staffing plan: Program directors serve full time in 

their position. Most have had at 
least some experience as bench scientists, 
but a sizable fraction return to research 
in a university or other setting after 
serving one or two years as a program 
director. 



Program Planning 



Sources of new 
program ideas: 



Mechanism for 
planning: 



Coordination: 



Research programs are clusters of pro- 
jects grouped by disciplines; but 
within a program, selected threads of 
inquiry may be emphasized. Care is taken 
to detect and nuture new threads of 
inquiry which are potentially of revo- 
lutionary impact. 

Every program director is responsible 
for spotting and stimulating new and 
important threads of inquiry, but in 
addition this responsibility is assigned 
full-time to a senior program director 
working at the institute director level. 

Areas of interest are coordinated with 
other agencies through informal contact 
at the program director's level. Pro- 
gram priorities are influenced by the 
director's office. 
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Program Development 

Sources of project ideas: 



Means of proposal 
review : 



Allocation of budget: 



Monitoring of 
performance: 



Scientists are funded to work on 
problems they have selected. Occa- 
sionally, a scientist's interests are 
guided by contact with a program 
director. Program directors advise 
applicants on the preparation of 
proposals. 

The program director selects a group 
of borderline proposals that are not 
clearly fundable or rejectable. The 
program director and the panel debate 
which sub^>3t of these borderline pro- 
jects is best to support along with the 
clearly fundable ones. 

In addition to working with his panel, 
the program director discusses priorities 
and opportunities with scientists during 
the year. At the biannual budget allo- 
cation time, the program director chooses 
the set of projects that his discussions 
and review panel session have convinced 
him offer the best combination of 
possibilities for revolutionary advance, 
normal science progress, and talent 
development in the lines of research 
being pursued. 

Substantive progress on projects is not 
closely monitored, though complete lack 
of progress (as indicated by progress 
reports) is grounds for termination. 
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Evaluation of outcome: No formal procedure. Reliance is placed 

on Including scientific reputation as an 
item of consideration in the proposal 
evaluation process. 

Program Evaluation 

Mechanism of evaluation: No formal procedures. 

ACTIVITIES 

There is considerable variation in the role played by the program 
director in Single Panel Review. In some sections, the panel and the 
program directors work as partners in the decision process with the program 
director acting as a decision maker and the panel serving in an advisory 
capacity. In other sections, the panel exercises much more decisionmaking 
authority, especially in the selection of projects and replacements for 
panelists whose term of service has expired. To get the greatest diffe- 
rence between Dual Panel Review and Single Panel Review, the paradigm 
presented below describes the former situation where the program director 
and the panel are in partnership. 

As in Dual Review, some Program Planning effort is exerted in Single 
Review. However, somewhat different methods are used. A primary diffe- 
hrence is less use of conferences and workshops. 

Program Planning 

i As in the Dual Panel Review paradigm, programs are a cluster of pro- 
jects grouped by scientific discipline, and not collections of logically 
interrelated projects deduced from a set of overarching goals. Thus, 
i"fields of inquiry" might be a better name than "programs" for these units 
of activity. Especially important new ideas that seem likely to develop 
into a distinctive field of inquiry are nurtured by the program staff 
through the usual means; traveling, workshops, and persuasion. Some 
responsibility for directing and encouraging these new ideas lies with 
jthe program directors, each of whom has responsibility for one field of 
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inquiry. But in addition, a senior program director reporting at the 
top management level spends all his time searching out and stimulating 
nascent fields of inquiry which offer significant potential. His prin- 
cipal means of stimulation are the same as the program director's. 

Program Development 

At intervals throughout the year, the program director visits 
scientists in his field of inquiry, discussing new trends, important 
events, and his own and others' opinions. These discussions help the 
program officer form his scientific priorities, and keep him informed. 
These discussions are useful for stimulating research proposals. 

Twice during the year the program director sorts a batch of perhaps 
100 proposals that have been submitted by individual investigators into 
three piles: the obviously inferior ones; the obviously superior ones; 
and twenty to thirty marginal proposals, some of which can be supported 
and some of which cannot. The program officer sends the twenty or thirty 
marginal proposals to his panel, and insists that they read them. He 
then calls a meeting of the panel to discuss the twenty or thirty proposals. 
Among the types of proposals that might be considered at such a taeeting 
are an off-beat idea that the project officer has not seen before, or 
ideas that appear to be outdated, or the work of a new Ph.D. The program 
director and the panel debate the merits and demerits of each proposal , 
not necessarily in order. The panel does not assign a score to the pro- 
posals, but tries to inform the program director of their opinions and 
information. The panel may also glance at the "clearly fundable" proposals 
passed by the program director. 

After the meeting, the program director selects the package of pro- 
posals that fits his budget and he believes to be an "optimal" balancie of 
innovative but risky ideas, normal science progress, special opportunities, 
and chances for development of new research talent in the lines of research 
he is pursuing. The budget of some proposals may be altered to eliminate 
weak portions, or increase strong ones. The program director negotiates 
with the applicants over these changes. The program director's final choice 
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is then reviewed and approved by his organizational superiors. 

Grants are awarded for a fixed amount expendable over a set period 
of time, usually less than five-years. During this time, annual progress 
reports are submitted; but unless major difficulties arise, the inves- 
tigator is free to choose his course. Project outcomes are net evaluated 
on a formal basis. 

A description, of this Program Development process appears in Figure 3. 

ORGANIZATION 

Structural Relationships 

Each program director has responsibility for one field of inquiry 
and runs his own panel. Several of these fields of inquiry are grouped 
together into an organizational unit called a section . Typically, a 
section has the name of a discipline, and each of the programs in it is 
a field of inquiry In the discipline. . Several sections together consti- 
tute a division, the second highest level organizational unit in NSF. 
This structure is diagramed in Figure 4. 

Staffing Relationships 

Program Directors . With rare exception, all program directors have 
a Ph.D. or equivalent and research experience. All serve full-time, but 
a sizable fraction have temporary appointments.' These program directors 
serve for a year or two and then return to their research environment. 
The rest of the program directors serve for longer terms. Program directors 
are selected by their immediate supervisor. 

Panelists . Panelists are selected by the program director in consul- 
tation with his supervisor and approved by higher echelons within NSF. 
The panelist's term of office is nominally two years, but there is no 
statutory requirement. The fields of inquiry are delineated so that all 
panelists can be specialists in a common research paradigm, but have 
different experimental or methodological approaches. Panelists are not 
paid for their services. 
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IV. MAIL REVIEW 



0\yERVlEW 

Replacing the peer panel in the Single Review paradigm with an ad 
hoc selection of reviewers who provide written comments via mail creates 
a third research management model, which will be called Mail Review , Mail 
Review is used in many places in the federal government, but particularly 
in the Engineering and the Mathematical and Physical Sciences divisions of 
NSF. Their experience with it extends back to World War II when many of 
the NSF managers now using it worked in the Office of Scientific Research 
and Development (OSRD) • OSRD used Mail Review somewhat differently from 
the way NSF does now. In OSRD good proposals were accepted, marginal ones 
were mailed to scientific peers for comment, and poor ones were dropped. 
In NSF every proposal receives mailed comment. 

In its basic form Mail Review differs only slightly from Single Panel 
Review, especially with regard to the program officer's responsibilities. 
An extended version of the basic model, however-, has evolved in NSF. The 
extended model differs from the basic model in two ways: (1) there are 
two rounds of Mail Reviews, and (2) after the project awards have been made, 
a panel is convened to review the list of awards. The first innovation was 
added as a means of letting each reviewer have the benefit of seeing the 
comments made by all the other reviewers of a proposal before he writes his 
fln;il opinion. The second innovation is a device for Program Evaluation. 

The essential features of this extended form of Mail Review are: 

o Support is given to individuals not to institutions to work on 
a problem they have, proposed. 

o Support is awarded for a limited period of time^ but can be 

renewed through presentat>ion and approval of a new application. 

o Proposals are evaluated by obtaining mailed comments from^ a 
number of scientists expert in the area of the proposal. 

o The majority of managerial effort is devoted to selecting a 

set of projects thought to optimally balance the various lines 
of research being pursued. 



This version was invented and used by Wayne Gruner, now Senior Staff 
Associate, Research Directorate, National Science Foundation. 
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o Programs are evaluated by periodically convening a panel that 
reviews the list of active projects. 

General Characteristics 

Primary output: Additions to fundamental knowledge about 

basic processes or practical problems. 

Mechanism of support: Finite duration project grants are made 

to individual research scientists. 
Generally, these scientists are employees 
of a university or non-profit firm. A 
small amount of formula support is awarded 
to the grantee's institution. 

Managerial empliasis: The most managerial effort is devoted to 

selecting the set of projects thought by 
program management to optimally balance 
the lines of research being pursued 
including prospects for normal science 
progress and revolutionary discovery. 

Staffing plan: Program directors serve full-time in 

their position. Most have had at least 
some experience as bench scientists, but 
only a portion return to research in a 
university or other setting, after serving 
as program directors. 



Research programs are clusters of projects 
grouped by disciplines; but within a 
program, selected threads of inquiry may 
be emphasized. Attempts are made to 
identify and develop potentially new 
threads of inquiry. 



Program Planning 

Sources of new 
program ideas: 
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Program Development 



Sources of project 
ideas : 



Means of proposal 
review : 



Allocation of budget: 



Monitoring of 
performance: 



Scientists are funded to work on 
problems they have defined. Occa- 
sionally, a scientist's interest in 
a program area is guided by contact 
with a program director. Program 
directors advise applicants on the pre- 
paration of proposals. 

Tiie program director sends each poten- 
tially fundable proposal to a few or 
more specialists in its field of inquiry. 
The returned comments are complied and 
sent back to each reviewer for additional 
comment . 

The program director discusses priorities 
and opportunities with scientists 
throughout the year. He also interacts 
with the audit panel convened to review 
his program. At the biannual budget 
allocation time, the program director 
chooses the set of projects that his 
discussions and previous review panel 
sessions have convinced him offer the 
best combination of possibilities for 
revolutionary advance, normal science 
progress, and talent development in the 
lines of research being pursued. A great 
deal of adjustment is often made in indi- 
vidual proposals. 

Substantive progress on projects is not 
closely monitored although complete lack 
of progress (as indicated by progress 
reports) is grounds for termination. 



ERLC 
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No formal procedure. Reliance Is placed 
on Including scientific reputation as an 
item of consideration in the proposal 
evaluation process. 



A permanent panel meets for two or three 
days to review the project selection deci- 
sions made by each program officer. The 
format is a roundtable discussion where 
panelists ask questions after a short 
presentation by the program director. Each 
panel reviews all the projects in a given 
section, which may comprise roughly a 
half-dozen programs. 

A review session is held biannually, 
shortly after each period during which 
projects grants are awarded. 

Program directors listen to the comments 
of panelists and incorporate their 
suggestions in future allocations. 

ACTIVITIES 

The Mail Review paradigm does not differ greatly from Single Review 
except in the way that the panel is used, so much will be a repetition of 
material presented in the Single Review sections. The principal difference 
is that in Mail Review the program director does not work with a panel in 
making project selection decisions. Because of this fact, there is less 
deviation from the Mail Review paradigm in NSF among various research 
programs* than is the case in Single Review. 



Except that the evaluation panel is not always used. 

ERLC 



Evaluation of 
outcomes: 



Program Evaluation 

Mechanism of 
evaluation: 



Timing of 
evaluation: 



Implementation of 
resxilts: 
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Program Planning 

As in the Single Panel Review paradigm, programs are a cluster of pro- 
jects grouped by scientific discipline , and not collections of logically 

interrelated projects deduced from a set of overarching goals. Thus, "fields 
of inquiry" might be a better name than "programs" for these units of activity. 

Especially important new ideas that seem likely to develop into a distinctive 

field of inquiry are nurtured by the program staff through the usual means: 

traveling, workshops, and persuasion. Some responsibility for detecting and 

encouraging these new ideas lies with the program directors; but in addition, 

a senior program director reporting at the top management level spends all his 

time searching out and stimulating nascent fields of inquiry which offer 

significant potential. His principal means of stimulation are the same as the 

program director's. 

Program Development 

At intervals throughout the year, the program director visits scientists 
in his field of inquiry, ciscussing new trends. Important events, and his own 
and others' options. These discussions help the program officer form his 
scientific priorities, and keep him informed; but also serve to stimulate 
research proposals. 

Proposals are received in the program director's office throughout the 
year from individual investigators. The program director sorts through the 
stack of received proposals and selects those that seem to qualify for fund- 
ing. Each selected proposal is sent to four or five specialists for a 
written review of its merits. The specialists are chosen to match the sub- 
ject area of the proposal and are given guidelines on the important criteria 
to consider in writing their review. The program director selects these 
reviewers from a list kept for the purpose. If the proposal is complicated 
or receives mixed reviews initially, additional first round reviewers are 
included. The first round of replies are compiled and mailed to each first 
round reviewer, who then revises his original review and sends it back to 
the program director. Upon receiving all the second round reviews, the 
program director has an expert, external assessment of a proposal's worth. 

Twice a year, the program di'^actor examines the backlog of reviewed 
proposals and selects a subset of them which fits his budget and V7hich he 
believes to be an "optimal"' balance of innovative but risky ideas, normal 
^ science progress, special opportunities, and chances for development of new 

ERIC 
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research talent in the lines of research he is pursuing. The budget of 
some proposals may be altered to eliminate weak portions, or increase strong 
ories. The program director negotiates with the applicants over these changes. 
The program director's final choice is then reviewed and approved by his 
organizational superiors. 

Grants are awarded for a fixed amount expendable usually over a set 
period of time, usually less than a five-year period. During this time, 
annual progress reports are submitted, but unless major difficulties arise, 
the investigator is free to choose his course. An investigator who is not 
making reasonable progress is given one year's notice prior to a final deci- 
sion to revoke his grant, but this occurs infrequently. Project outcomes 
are not evaluated on a formi^. basis. 

A flow chart of this process appears in Figure 5. 

Program Evaluation 

Programs are evaluated by a panel of scientists, specifically appointed 
for the purpose. The panel convenes shortly after each period when project 
awards are Tnade to review the list of projects selected and rejected, and 
cross-examine the program directors on their decisions. The panel evaluates 
all the programs managed by a section, which in NSF's case means half a 
dozen or fewer programs. 

The meeting format is not highly structured, but is rather informal* 
The chairman of the panel, elected by the panel members themselves, asks 
each program director in turn briefly to describe his decisions. Following 
this presentation, the meeting is opened for roundtable discussion. The 
attitude of the program staff is to "throw open the files, and let them see 
anything they want." Back and forth discussion of each program proceeds 
for a while until all opinions are on the table. The panel can vote a 
resolution if desired, but usually the spoken comments of the members are 
deemed adequate evaluations. 

These evaluations are reflected in future decisions only if the program 
directors wish to follow them, but clear signals are difficult to ignore. 
Scientists' opinions about the abilities of a program director easily find 
their way back to the program director's supervisors. In addition, the sec- 
tion Supervisor attends the panel meeting, and talks with the panelists on 
an individual basis whenever the need arises. 
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ORGANIZATION 

Structural Relationships 

The same organization chart as for Single Panel Review applies. 

Staffing Relationships 

Program Directors . With rare exception, all have Ph.D.s or equiva- 
lent. The ^ rate is one departure per approximately two positions 
for half of the staff, and one departure per approximately eight years 
for the rest* Each program director is selected by his immediate super- 
visor. 

Mail Reviewers . Mail reviewers are selected by the program director 
with the objective of obtaining a match between topics in the proposed 
research and specialties covered by the reviewers. The list from which 
reviewers are drawn is frequently updated, and so long that only rarely 
does any reviewer see more than three proposals per year. A great deal 
of work must be exerted to build and maintain the list of mail reviewers. 

Panel. With Foundation management approval required, the program 
directors and their supervisors select replacement panelists from the 
community of scholars. The term of office is two years, but there is no 
statutory requirement. The panelists span a greater range of research 
interests than the programs evaluated and may include applications-oriented 
researchers. Panelists are not paid for their services. 
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V. "NO" REVIEW 

OVERVIEW 

During the decade just after World War II, the lead agency in support 
of basic science in the United States was the Office of Naval Research 
(ONR) in the Department of the Navy. ONR hsad inherited this role from the 
Office of Scientific Research and Development (OSRD) . OSRD had sponsored 
several large projects, such as the Manhattan project, and had functioned 
as a presidential level center for mobilizing scientific manpower in sup- 
port of the war effort. To many, OSRD's results proved the importance of 
scientific research to national security, and the need for federal support 
of science. While agreeing with the need for continuing federal support, 
many scientists felt uneasy about having this support come through the 
military, and campaigned actively for support through an independent agency. 
This effort, which was lead by Vannevar Bush, culminated in the creation 
of the National Science Foundation in 1950. In the meantime, however, 
science needed monetary support or the large effort built up during the war 
would have had to be disbanded* A coalition of forces managed to achieve 
this in 1946 by creating ONR and providing It with the needed funds. ONR 
used them to build strong basic research programs. By the middle 1950s, 
NSF had grown large enough to be the lead science agency, and ONR began to 
shift its emphasis to research on specific Navy needs. 

Throughout its history, and especially during the basic science era, 
ONR has employed a distinctive form of R&D management. Its program officers 
have a great deal more autonomy and responsibility for building their pro- 
gram than most program officers, and they stress attracting the best research- 
ers to their program, not choosing the best projects by evaluating proposals. 
ONR's philosophy amounts to putting its money on people rather than projects. 

The program officer's mode of operation has been to consider the needs 
of the Navy and then identify the areas of science most relevant to those 
needs. Within each area, the program director tries to find the most capable 
researchers and add them to his program. This approach to R&D management is 
a modified form of the philosophy followed In OSRD. 

*Schaffer, Dorothy, The National Science Foundation, Praeger, New York, 
1969, Chapter I. 
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It was natural that ONR adopted these policies, since much of its budget 
and staff came from OSRD when ONR was created, ONR went further, however, 
in relying on the program officer for decision making and in not using formal, 
external advisory mechanisms. NSF, which obtained most of its initial staff 
from ONR, has gone in the other direction, since it incorporates more outside 
advice in the management process than either ONR of OSRD. 

While program directors rely little on formal review procedures, they 
expend a great deal of effort Tuaintaining an informal network of contacts in 
the scientific community. These sources are used as a means of detecting new 
idpas, finding new researchers, and deciding which researchers to add to their 
program. Because review is not done formally but informally through this net- 
work of contacts, ONR's system is called No Review with the qualifier "No" 
in quotations. 

Over the years, the ONR directorship together with its divisions have 
evolved variants of the basic management philosophy, but the purest form, 
used when ONR. had responsibility for fostering basic science, will be 
described here. It relies least on formal mechanisms for linking advice 
from the outside into the management process. Unlike the earlier times, 
when all of ONR's budget came from the Navy's line item R&D appropriation, 
much of the budget today is obtained from operating arms of the Navy and 
from the Office of the Secretary. Each sponsor has different types of needs 
and, consequently, ONR has had to modify its original method of management 
to fit these new demands. 

In its pure form, the essential features of the ONR paradigm are: 

0 Support is given to individuals ^ not to institutions, to work on a 
problem they have proposed. 

0 An understanding is reached with an investigator that he will be 
supported for an indefinite period of time, contingent on contin- 
ued productivity. 

0 directing and attracting the most capable researchers to the agency 
program receives the most managerial effort. 



SUMMARY 



\ 



General Characteristics 



Primary output: 



Additions to knowledge about basic processes « 



Mechanism of support: 



A finite duration contract is awarded to 
individual research scientists for level 
of effort support of salary and equipment 
expenditures, though a promise of contin- 
uing support is understood. Generally, 
these scientists are employees of a univer* 
sity^ research institute, or sometimes a 
firm. 



Managerial emphasis: 



Finding and keeping a collection of the most 
capable researchers in fields of inquiry 
relevant to agency needs requires the bulk 
of management effort. 



Staffing plan: 



Program directors serve full-time in their 
position. Most have had at least some 
experience as bench scientists but only a 
few return to research after serving as 
program directors, A good portion of the 
program directors are young Ph.D.s, 



Program Planning 

Sources of new Research programs are collections of the 

program ideas: ^^^^ ^^^^^^ funded, grouped by scientific 

disciplines. Program directors are con- 
stantly on the look-out for new ideas which 
arise in the scientific community, which 
are interesting to the program director, 
and which seem important to science and 
the agency mission. 
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Mechanism for planning: 



Coordination: ' 

Program Development 

Sources of project ideas: 



Means of proposal review: 



Allocation of budget: 



Monitoring of performance: 



ERIC 



The most creative and capable people in 
new areas are sought and encouraged. 
Emphasis is placed on attracting young 
Ph.D.s. 

Areas of interest are coordinated with other , 
agencies through inforroal contact at the 
program director's level. 

Project ideas per se are not important to 
the allocation process, but to qualify for 
support at renewal time scientists do sub- 
mit a statement of work which indicates 
topics to be pursued. 

Other than a site visit by branch office 
personnel to check on research facilities, 
and a reading of proposals by the program 
director, there is no formal proposal 
review procedure. 

The program director discusses priorities 
and opportunities with his contracted 
scientists and others throughout the year. 
Discussions are also held with managers 
in operating environments. As opportuni- 
ties come along, the program director 
allots his budget of funds to "optimize" 
the levels of effort in his program. 

The program director makes at least one, 
and often two site visits per year to each 
scientist supported. Discussions cover 
the trend and significance of results 
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obtalned by the scientist and by other 
scientists, and problems encountered. In 
addition an annual "short form" progress 
report is submitted. 

Evaluation of outcomes: Performers are not evaluated on a formal 

basis, but each program director keeps up 
to date on who makes the discoveries in 
his area of responsibility; 

Program Evaluation 

Mechanism of evaluation: Not done formally. 

ACTIVITIES 

In the "No" Review paradigm no specific mechanisms are used for program 
planning although management emphasizes the necessity of maintaining surveil- 
lance for new program possibilities. The policies used to implement this 
responsibility are described in the following section, after a brief discussion 
of the nature of the research activity managed. 

Program Planning 

As in Dual Review, Single Review, and Mail Review paradigms, research 
programs are not logically interrelated sets of projects deduced from over- 
arching goals. In this paradigm the emphasis is on people, not projects. 
The method of management is to select the fields of inquiry which are most 
likely to produce results of importanv to the sponsoring agency, and then 
attract the very best people in these fields to the agency's programs. As 
a result, programs are more accurately reviewed as groups of people with 
similar interests rather than collections of projects with logically inter- 
related tasks. 

The groups of researchers are labeled by scientific discipline or sub- 
discipline since the individuals in each group have selected problems and 
are using methods ot solution which identify them as members of a particular 
scientific discipline. The grouping of programs by disciplines facilitates 
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the management process since the program director's principle m^ans of manage- 
ment Is tapping Into the Invisible college to which his researchers belong.- 

The program director is held responsible for detecting the significant 
discoveries which portend major changes in his field of responsibility. He 
does this by knowing almost everyone :ln his field reasonably well, and 
talking with them often. When a significant new discovery comes along, the 
program director learns of it through these long-established personal contacts. 
If he decides that the discovery Is important to the agency's mission, his job 
Is to attract the very best people in this line of Inquiry to his program. 

With this method great teliance is placed on the program director for 

determining what new directions the research program will take. It Is his 

responsibility to assess agency needs and find the ideas relevant to those 

needs. His Interests, and his Judgments in large measure determine what new 

directions will be pursued. ONR's Research Management Guide says that: 

A program can be designed to strengtbpn those scientific sub- 
fields indent if led to be of special concern to future service 
needs. A program can also be effective as a means of creating 
a focus, or even estabj^ishlng a new subfield when it is lack- 
ing in the discipline. 

Compared to Dual Review, Single Review, and Mail* Review methods of man- 
agement, this paradigm relies less on proposal pressure for setting the allo- 
cation of money to new areas. Instead it depends on the program director's 
choice. Whether this results in the system lagging or leading scientific 
programs depends on the program director's abilities. 

To the extent necessary, program directors coordinate rheir programs 
by keeping in personal touch with their counterparts in other agencies. Over 
time an understanding is reached about the topics each party will emphasize. 
Project by project coordination is not necessary since the scientific com- 
munity gives no credit for duplicated effort. 

If the program director's supervising managers see an area or new idea 

which seems important, but is being improperly supported they will add a new 

program director who is interested in this area to the staff before they 

will interfere with current programs. In the words of one division director: 

In general, we won't have a program unless a staff member of our 
group is professionally and deeply concerned with that area. We 



U.S. Office of Naval Research, ''Research Management Guide," Department 
of the Navy, Washington, D.C., 1969, p. II-9 
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would not decide by flat for a program In biophysics unless we 
had a physicist on the staff Interested In the area, visiting 
campuses, and drumming up support from Investigators.* 

Program Development 

Program Development Is managed with the same general philosophy as Pro- 
Gram Planning. In fact, by the nature of this paradigm It Is difficult to 
distinguish between Program Planning activities and Program Development 
activities* 

The responsibility of each program officer Is not to "take proposals 

t** 

and score them, but to develop programs. He does this by maintaining a 

*** 

really cogent grasp of his scientific or technical field," and knowing 

who are the best people In his field. His chief responsibility is to attract 

these researchers to his program. The way this is done has been described as 

follows by a division director: 

Informally, we use the whole scientific community. There is a lot 
of travel. We visit contractors and non-contractors. We visit 
most of the professional society meetings, and have extensive con- 
versations regarding research. The use of the scientific community 
is not for the review of individual proposals but with respect to 
individual investigators. At the time I was at the branch level, 
there were probably not five people in my field not known person- 
ally to me. 

Each researcher supported by the program director is site-visited once 
and usually twice per year. The program director finds out about new results 
the researcher has obtained, new results obtained by others and the importance 
of all these results to scientific progress. By continually checking with 
his researchers, and others too, the program director keeps aware of who is 
making the Important contributions in his field of responsibility. 

Particular attention is paid to monitoring the crop of new doctoral 
graduates. By talking to his network of contacts the program director can 



Rettlg, Richard, "Federal Support of Scientific Research, a Comparative 
Study," unpublished Ph.D. thesis, Massachusetts Institute of Technology, 
August, 1967. 

Rettlg, op. clt., p. 90 

***U.S. Office of Naval Research, "Research Management Guide," Navy 
Department, Washington, D.C., p. I I- 2. 
**** 

Rettlg, op. clt., p. 97 
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find who the best students are and what they have done. Then he arranges 
appointments and finds out what each student's Interests are. At a later 
time, the program director Informally suggests to some of these graduates 
that they should submit request for support. 

The program director stands a good chance of attracting these gradu- 
ates, or anybody else he wants, becuase he offers the likelihood of long 
term support. The program director can promise continuing support to In- 
vestigators, because he has been delegated great authority over the expendi- 
ture of his budget as a concommitent of responsibility for getting the best 
people Into his program. Indefinite support is not guaranteed, however, 
sfmce that would reduce incentives for performance and limit the possibilities 
for changing program direction to capture new opportunities. He makes support 
contingent on continued productivity and mutuality of interests. 

Contracts with researchers last for a finite period, renewable In compe- 
tition with other prospective contractors eligible at the same time. Each 
prospective contractor submits a proposal, but formal requirements are 
minimal. The proposal should state the Investigator's qualifications, his 
facilities for research, a brief statement of research objectives and scien- 
tific methods to be employed, and a project budget. 

As the first step In reviewing proposals, the closest ONR regional 
office sends an officer out to check on the proposing investigator's research 
facilities, and work environment. The program director reads this report, 
and the Investigator's proposal; and then decides whether he will support 
the investigator and at what level. He often discusses this decision with 
his branch chief and with other program officers. Awards are not made at 
predetermined times, but occasionally as opportunities arise or needs become 
great. 

In making his allocation the program director is cautioned against 

following the advice of his contacts against his judgment of what is best 

for agency needs or science In larger perspective. The Research Management 

Guide says that the program director: 

should be wise In supporting fields of special naval relevance 
that may not be currently popular. To effect this he should 
maintain personal contact with the recognized leadership in 
the discipline, ... but must also recognize that science has 
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its fads and fancies and followers thereof. The program director 
must avoid becoming a 'follower' by steering clear of intimacy 
with any one camp. Indeed, it is almost always better for the 
ONR program director to rely, in the last analysis, on his own 
educated and open-minded judgment with the best interests of 
the Navy at heart, than to eely solely on biased peer-grouped 
(sic) judgments,* 

A tabular description of the Program Development process appears in 
Figure 6. 

Program Evaluation 

No formal mechanism for Program Evaluation is employed; but on an in- 
formal basis, the branch director is responsible for maintaining the quality, 

if if 

balance, and productivity of all the programs under him. He does this 
through continual interaction with his program directors. The branch chief 
allocates his budget allotment to the program directors, 

ORGANIZATION 

Structural Relationships 

An organization chart for ONR is shown in Figure 7. 

Staffing Relationships 

Program Directors , With rare exception all have Ph,D,s or equivalent. 
All serve on a full-time basis, although half are young Ph,D,s, and stay only 
for a short time. More often than with other management models, research 
experience is a qualification for the program director's position. Program 
directors are selected by their immediate supervisor. 



Office of Naval Research, "Research Management Guide," Department of 
the Navy, Washington, D.C, , 1969. 

Rettig, op. cit,, p. 90, 
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Appendix A 
STRUCTURE OF FUNDAMENTAL RESEARCH 

John Wirt 

INTRODUCTION 

This appendix is a paraphrased sunnnary of T. S. Kuhn's well-known 
work, the Structure of Scientific Revolutions^** Among all the models 
of the scientific research process examined, Kuhn's was found to be the 
most penetrating and operationally useful. It describes differences 
between the various stages of the fundamental research process and speci- 
fies mechanisms through which scientific progress is made. Knowing these 
factors, management . can be more precise in setting conditions that promote 
progress, and better understand what results should be expected from funda- 
mental research. Thus, Kuhn's model provides conceptual tools that aid in 
thinking about what procedures to use in managing fundamental research and 
when to use them. 

Kuhn eschews the simple model that fundamental research produces an 
inexorable, steady accretion of proven propositions. His evidence is that 
historical facts do not fit with an "accretive" model of scientific research. 
In attempting to chronicle science as an incremental process, science his* 
torians suspect that these questions, which are relevant only if science 
develops by the accumulation of Individual discoveries and inventions, are 
the wrong sort to ask. Simultaneously, these same historians confront 
growing difficulties in distinguishing the "scientific" component of past 
observations and beliefs from what earlier historians of science had 
readily labeled "error" and superstition." The more carefully they study, 
say, Aristotelian dynamics, phlogistic chemistry, or caloric thermodynamics, 
the more certain they feel that once current views of nature were, as a 
whole, neither less scientific nor more the product of human Idiosyncrasy 



Any inadequacies or inaccuracies of interpretation are the author's 
responsibility. 

Kuhn, T. S., The Structure of Scientific Revolutions^ International 
Encyclopedia of Science, Vol. II, No. 2, Univ. of Chicago Press, Chicago, 




1970. 



-48- 



than those current today. If out-of-date beliefs are to be called myths, 
then myths are still possible in scientific knowledge because the same 
methods used conducting science in the past are still employed tod^y. If 
out-of-date beliefs are to be called products of science, then science 
has included bodies of belief quite incompatible with what sci(=tnce holds 
today. Given these alternatives the historian of science is compelled, 
says Kuhn, to choose the latter alternative, and conclude that out-of- 
date theories are not in principle unscientific because they have been 
discarded. This choice, however, makes it difficult to see scientific 
research as a process of accretion alone, but also revolution where past 
accomplishments are discarded in favor of new accomplishments. 

Problem Solving iu Fundamental Research 

The conventional model is that scientific knowledge is embodied in 
theories, laws, and rules, and that students learn a scientific field by 
learning these theories, laws, and rules. The conventional model is also 
that students learn to apply these fundamentals to practical cases by 
solving concrete problems. Kuhn asserts that this localization of the 
cognitive content of science is misplaced. Students do not learn laws, 
or theoris, or rules in the abstract, and then how to apply them. Rather, 
Kuhn says, a student learns a scientific field by learning the solutions 
to problems. Whether or not a student learns laws and rules in the abstract 
is debatable, for even scientists, claims Kuhn, are little better than laymen 
at characterizing the bases of their field. The solutions that students 
learn are the specific examples encountered from the start of scientific 
education, including solved problems in textbooks, problems at the end of 
chapters, laboratory e.iperiments, and Journal publications. Kuhn calls 
them exemplars . Given a new problem the student finds its solution by 
perceiving similarities between problems he can solve and the problem he 
wants to solve. 

After solving enough problems some students are able to achieve a 
higher skill, which is the ability to compose new problems that are like 
ones already solved, but requiring innovation to solve them. Kuhn asserts 
that students never learn rules for solving problems, or selecting new 



problems 9 or formulciting new problems; yet with study students gain an 
ability to pose new problems In their field that, when solved, are a 
contribution to knowledge. Kuhn calls these problems puzzles for their 
analogous characteristics with more familiar puzzle forms. 

When a student attains this ability to pose and solvfi significant 
puzzles, he qualifies as a member of the scientific community with a 
speciality in the field of inquiry he has mastered. The process of acquiring 
knowledge of and the ability to extend scientific laws and conceptc^ by 
learning the solutions to exemplary problems Kuhn and others call "learning 
by doing. " 

Knowledge Paradigm 

Thus, Kuhn argues that the set of exemplars is one of the principal 
elements of knowledge in a field of inquiry. Exemplars define the domain 
of a field of inquiry, determine the rules that govern the formulation of 
new problems, specify acceptable forms of solutions, and express the 
scientific laws that have been discovered. Exemplars function for researchers 
in a field of inquiry in much the same way that court decisions determine 
rules and law for judges practicing the common law tradition. The 
exemplars are the points of agreement on. fund&jnental entities and interactions 
which a scientific group must have to guide its way deeply into the com- 
plexities of nature. It is in this sense that exemplars are a tacit expression 
of the state of knowledge in a scientific field of inquiry. 

In addition to exemplars, Kuhn has identified three other elements of 
tacit knowledge in a field of inquiry. These are that the members of a 
scientific community in a field of inquiry: 

o Have a shared commitment to certain beliefs — such as the 

molecules of a gas behave like tiny elastic billiard balls or 
that certain kinds of equipment should be used for experimen- 
tation; 

o Agree on the meaning of some symbolic representations — such 
as f=ma (which is operationally useless to someone who has not 
solved Newtonian problems) ; and 
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o Share a set of values — such as the appropriateness of imposing 
social concerns during problem formulation, the degree of sim- 
plicity demanded in theories, and tolerable margins for error 
in predictions. 

Altogether, Kuhn calls these elements a knowledge paradigm. These know- 
ledge paradigms bear the same relation to the scientific laws and rules in 
a field of inquiry as the management paradigms of this report bear to par- 
ticular philosophies of management. Kuhn asserts that it is not hard for 
researchers in a field of Inquiry to reach a consensus on what the essential 
elements in their paradigm are, but usually impossible to get a consensus 
on what the fundamental concepts, laws, and rules are. Kuhn goes even 
farther by suggesting that rules and laws do not exist in science, and gives 
some evidence. In any case rules and laws are not needed for scientific 
progress to occur. Kuhn concludes that knowledge paradigms are the funda- 
mental units in the scientific research process. 

The size of a paradigm can be gauged by noting the number of researchers 
sharing it, which means the number who know the paradigm and use it in their 
work. In most cases, less than 100 researchers actively share a paradigm 
at any one point in time. Usually, these researchers will come from a variety 
of scientific disciplines and subdisciplines. The scientific community can 
be partitioned at the most aggregate level by categories such as natural 
scientists and social scientists; and at a finer level by disciplines such 
as physicists, chemists, astronomers, and others. At the next finer parti- 
tion, the groups can be labeled by subdisciplines; organic chemists, solid- 
state physicists and so on. In most cases there are substantially more than 
100 researchers in these categories. Thus, the community that shares a para- 
digm is at an even finer level of partition than subdisciplines groups. And, 
since the community sharing a paradigm is usually interdisciplinary in nature, 
the partition of scientific activity into paradigms produces groupings of 
scientists which intersect several of the discipline categories of scientists. 

One example of a research community having a shared paradigm is the 
phage group of molecular biologists and others, which has recently been studied 
in the history of science literature. Another exan^)le is the group of econo- 
mists who share the general equilibrium, perfect competition paradigm. Another 
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example of a paradigm is Newton's Theory of Motion. This paradigm has 
been replaced by another paradigm, Einstein's Theory of Relativity, 

Using the concept of a knowledge paradigm, Kuhn then presents a model 
for the dynamics of the knowledge development process. His model has 
four phases: 

o Normal Science, 

o Crisis, 

o Revolution, and 

o Preparadigm Research. 
Each of these activities will be discussed in section below. 

NORMAL SCIENCE 

The first phase of the knowledge development process Is normal science 
where the research activity is directed toward actualizing the promise offered 
by ^, new paradigm. Most scientists spend the great bulk of their lives in 
normal science pursuits. The attempt is to force nature into the preformed 
and relatively inflexible conceptual boxes that the paradigm provides. This 
is done by gathering facts that the paradigm displays as revealing, by in- 
creasing the match between fact and prediction by amending theory, and by 
further articulating the paradigm to extend its scope and precision. At any 
time a paradigm contains a core of "recurrent and quasi-standard illustra- 
tions" of theory and its application, but also a "penumbral" area of specu- 
lations and achievements whose status is still in doubt. An objective in 
normal science is to resolve these ambiguities and at the same time open up 
new ones for investigation. The researchers resolve these ambltuitles by 
solving the implicit puzzles posed by the ambiguities and framed by the shared 
paradigm. Concurrently, however, new ambiguities arise from apparently 
arbitrary factors, compounded of personal and historical accident which Is 
always a formative ingredient in normal science progress. But, normal 
science is inherently a narrowly construed activity • The paradigm forces 
researchers to investigate a part of nature in a detail and depth otherwise 
unimaginable. There is no attempt to probe for unexpected novelty either 
conceptual or phenomenological for such a discovery could possibly upset 
the paradigm on which normal science depends for guidance. The game is to 
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prove the expected. /i.iy search for fundamental novelties Is resisted. 
Unlike the other phases of fundamental research, normal science is cumu- 
lative in nature. 

While the normal science does not aim at novelty, paradoxically, its 
result is almost invariably to expose anomalies between paradigm predictions 
and fact. Paradigms which do not reach this stage after an extended period 
of time are gradually dropped by the scientific community and assimilated 
into engineering. The paradox is that in achieving greater scope and pre- 
cision with normal science activity, it becomes simultaneously easier to 
detect where theoretical predictions and fact are misaligned. The frequency 
with which simultaneous discovery occurs is testimony to the completeness 
of the normal science tradition and the way it prepared for change. 

CRISIS 

Appearance of anomalies in significant number signals the onset of 
crisis , another stage in fundamental research activity. Some anomaly always 
exists in normal science, for that is what makes puzzle solving a challenge. 
But, the presence of too many anomalies can raise a crisis. Crisis becomes 
acute when the complexity of a paradigm proliferates sufficiently to loosen 
the rules of normal puzzle solving, or increases faster than the paradigm's 
explanatory power. Crisis can also be heightened by external social pressures, 
as was the case near the end of the Ptolemaic system of astronomy, when 
difficulties with the calendar became increasingly irritating. 

Response to Crisis 

The responses to crisis are many and varied. The minimal response is 
to avoid the anomalies and work on other problems. Greater response is 
likely if the anomalies call a fundamental generalization of the paradigm 
into question, or if derivations of the paradigm are used in practice. A 
crisis state is marked by a turn to philosophical analysis, and debate on 
paradigm fundamentals. Articles expressing discontent often appear. Many 
resist the anomalies in the belief that old methods will prove adequate to 
resolving the differences and sometimes they prove correct. Others mount 
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Increasingly divergent theoretical and experimental attacks on the unsolved 
problems, exhibiting a willingness to try novelties not acceptable during 
normal science activity. Sometimes effort may concentrate on the attacking 
anomalies directly by doing research which exposes them more clearly. This 
activity generally results in novel discoveries unlikely to have occurred 
during normal science and proposal of ma^y radically different theories. 

Result of Crisis 

The final outcome of a crisis state varies, too. One outcome has 
already been mentioned, that the crisis is resolved by reworking the old 
paradigm. Other times the anomalies can be so severe that the paradigm is 
set aside. In this case the researchers who shared the paradigm will migrate 
to other fields. Another result is that a new paradigm appears and even- 
tually replaces the old one. 

New paradigms emerge from the sequence of novel discoveries and proposals 
of theory that are the scientific community's response to crisis. In the past 
more than a decade of crisis activity has often passed before the new para- 
digm emerged. Study of the atomic nucleus is probably in a crisis state at 
the present time, since the number of competing theories is getting larger, 
and elucidating experiments show greater complexity than expected. 

The emergence of a new paradigm is a complicated process and occurs 
over an extended period of time, contrary to the commonly accepted perception 
that scientific discoveries are traceable to a single event. Usually a ''complex 
of observation and conceptualization, fact and assimilation to theory are 
inseparably linked in discovery." The awareness of anomaly, a constellation 
of unrelateable facts produced by the work of many people, and an array of 
proposals for theory are often formative ingredients in the detection of a 
scientific novelty. Sometimes facts and theories previously overlooked may 
be rediscovered. Kuhn describes the process by which a paradigm emerges 
by using the metaphor of a psychological experiment where subjects were 
given short and controlled exposure to a sequence of playing cards. All the 
cards in the sequence were normally figured but some were miscolored. For 
example, one was a red six of spades. The subjects were not told of the 
anomalies. At short exposure durations all subjects identified the cards 
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without hesitation at their normal face value. As the exposure duration 
was increased, some subjects began to hesitate and display awareness of 
anomz-^ly. Upon further increasing the exposure duration these subjects 
identified all the cards correctly. A few subjects, however, were never 
able to make the requisite adjustment of their categories, even at 40 
times the average exposure needed to identify normal cards. Kuhn attributes 
the same set of characteristics to the emergence u£ a new paradigm: "pre- 
vious awareness of anomaly, the gradual and simultaneous emergence of both 
observational and conceptual recognition, and the consequent change of 
paradigm categories and procedures often accompanied by resistance. Fact 
and theory emerge simultaneously. 

REVOLUTION 

The scientific community has no calculus for determining whether or 
not to adopt an emerging paradigm (ox if it is the case, one of several 
paradigms) . Early versions of paradigms are rarely successful at solving 
all the anomalies which have arisen, and usually fail to solve all of the 
problems already solved in the old paradigm. New theories usually fit data 
poorly, and no objective measure for goodness of fit exists to provide a 
basis for comparison. Furthermore, and it is Kuhn's most difficult point 
to perceive, adopting a new paradigm entails a gesalt switch in the researcher •s 
view of the world. Thus, adherents to different paradigms are dealing with 
incommensurate entities and talk past each other on the subject of which 
paradigm should be used. 

The gesalt switch involves more than just a change in the way a set 
of data is interpreted. The model that different views of the world are 
only different interpretations of the same data can be traced to Descartes. 
Kuhn asserts that this paradTgTn is in a state of crisis itself due to results 
in philosophy, psychology, linguistics, and even art history but no alter- 
native paradigm for explaining perception has emerged. The difficulty of 
rejecting Descartes' paradigm is seen in Kuhn' s paradox, "that though the 
world does not change with a change of paradigm, the scientist afterward 
works in a different world." The history of science shows, says Kuhn, that 
scientists measure different variables and engage in a different set of 



-55- 



laboratory manipulations after a new paradigm is adopted. "What occurs 
in (adopting a new paradigm) is not fully reducible to a reinterpretation 
of individual and stable data ... Interpretive enterprise ... can only 
articulate a paradigm^ not correct it,** where correct means to transform it 
it into truth equivalent to nature. Paradigms are necessarily artifacts, 
albeit esoteric ones, that are "not corrigible by normal science at all. 
Instead normal science ultimately leads only to the recognition of anomalies 
and to crisis. And crisis is terminated, not by deliberation and inter- 
pretation, but by a relatively sudden and unstructured event like the 
gesalt switch. Scientists then often speak of the 'scales falling from 
the eyes' or of the 'lightening flash' that 'inundates' a previously 
obscure puzzle, enabling its components to be seen in a new way for the 
first time ... No ordinary sense of the term interpretation fits these 
flashes of intuition that are part of the paradigm-emerging process. Though 
such intuitions depend on experience, both anomalous and congruent, gained 
with the old paradigm, they are not logically or piecemeal linked to par- 
ticular items of that experience as an interpretation would be." Instead 
large portions of that experience are gathered, and transformed into a 
"rather different bundle of experience" and "thereafter ... linked piecemeal 
to the new paradigm but not to the old." 

Kuhn calls the process of rejecting an old paradigm and adopting r. 
new one a scientific revolution. As mentioned before revolutions occur 
gradually because the new paradigm is not clearly effective in rccolving 
the crisis, and requires a fundamental, qualitative shift in perception 
that scientists may not easily adopt. Appeals to the esthetic may be 
required. Furthermore, assimilation often requires that some former theory 
and fact be abandoned, that which is sometimes difficult to do. Thus, 
scientists do not switch rapidly to a new paradigm as it emerges. 

For all these reasons, a scientific revolution is not an additive or 
incremental occurrence. Eventually greater scope and precision in predicting 
nature will be achieved after a period of normal science activity, but a 
fundamental change of perception will have occurred and a different set of 
problems will be solved. Whether or not an absolute gain in knowledge is 
achieved is not a matter of analytic estimation. 
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PRE-PARADIGM SCIENCE 

The usual pattern in mature science Is successive transition from 
one paradigm to another via revolution over decade or decades-long cycles. 
The fields normally considered to be science, such as most of the natural 
sciences, display this pattern. 

Some fields of fundamental research have not reached this stage, 
particularly In the social sciences. Kuhn labels these fields as being 
In the pre-paradlgm stage of development, a stage In which all fields of 
fundamental research begins have begun. 

Pre-paradlgm research Is recognizable by the existence of debate on 
fundamentals, and whether or not the field Is a science. These debates 
serve to define schools of thought rather than solve problems. When the 
paradigm state Is reached, such debate ceases. Another clue to the pre- 
paradlgm condition Is the absence of the puzzle solving activity that Is so 
crucial to normal science. Lacking In agreement on fundamentals, methodo- 
logies, and instrumentation, pre-paradlgm scientists must always start their 
major expositions from a set of specified first principles. Activity Is 
characterized by a continual competition between a number of distinct views 
of nature all roughly compatible with the dictates of the scientific method, 
but not with each other. Fact gathering Is more random than In the paradigm 
stage and more limited to the wealth of data at hand. Technology and practice 
(like medicine) are more likely to be a source of data, since esoteric facts 
which could not have been so easily discovered by casual means may lie 
exposed. The transition to paradigm science has been made when specialized 
journals form, and laymen can no longer keep track of progress by reading 
original papers. Books will not be closely related to professional status 
either. 

IMPLICATIONS FOR FUNDAMENTAL RESEARCH MANAGEMENT 

Most of the management implications of Kuhn's model are explored In 
WN-7680, where R&D management methods are evaluated. Some points, however, 
can be made here. 

i/irst, the role of exemplars In paradigm development makes the utili- 
zation of scientists or at least scientific opinion in project selection 

ERIC 
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processes imperative. The logic of Kiihn's argument is that there is no 
other way to maintain the integrity of normal science progress unless 
those making the project selection decision know the ^^.ock of solved 
problems. The role of exemplars also suggests strongly that in managing 
fundamental research, reliance should be placed on unsolicited proposals 
from scientists as the source of project ideas. 

A second aspect of Kuhn's models of importance to management is that 
sudden, dramatic discoveries of practical importance will not often appear, 
nor will the research process respond to shifting practical priorities* 
The emphasis is more on the esoteric, and penetrating exploration of pro- 
blems which previous research implies can be solved than timely response 
to pressing practical needs. Promising approaches are pursued to great 
depths of detail though they are of rather narrow concern. Progress comes 
in large spurts coinciding with the occurrence of scientific revolutions. 
But, these happen on a scale cf decades, not years, so management needs to 
be patient. 

Another key implication of Kuhn's model is that disciplines are not 
the appropriate categories for distinguishing among scientists when placing 
them on panels or using their advice in other ways. A more useful cate- 
gorization would be based on paradigms, since a group of scientists knov/ing 
the same paradigm yet having different discipline backgrounds will be 
able to communicate better with each other and make sounder judgments in 
their field than a group o£ scientists all from the same discipline but 
adhering to different paradigms. 



-58- 



Appendix B 
ASSESSMENT WORKSHOPS 

INTRODUCTION 

The National Institute of Dental Research has devised ''Assessment 
Workshops,** as a means of evaluating and revising research programs. Assess- 
ment: workshops are formal In procedure, but largely Indirect In their effect 
on the research community. 

DOMAIN OF A WORKSHOP 

The assessment workshop technique Is Intended to deal with research 
paradigms of small size. Examples on which NIDR has assembled panels are 
"Genetics of the Cleft Palate,*' and "Dental Amalgam" (the material used to 
fill teeth). Typically, there are fifty to one hundred researchers worldwide 
associated with the paradigm, which is less than the four hundred or more 
typically associated with basic science paradigms. Another gauge of the size 
of these research paradigms Is that NIDR expects that each program director 
will conduct one or two assessment workshops each year oa paradigms In his 
area of responsibility, returning to each paradigm about once each five 
years. The NIH organization chart In the Structural Relationships section 
of the Dual Review Paradigm description lists the areas of responsibilities 
that program directors have In NIDR. 

PANELISTS 

About fifteen panelists are chosen by the program director. Partici- 
pants are very carefully selected to achieve representation of: 

0 all of the significant research approaches to the paradigm, 
0 practitioners who have reputation for quality research on the 

practical derivatives of the paradigm, 
0 foreign researchers (if they have contributed). 



For a definition of research paradigms, see Appendix A. 
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0 industrial researchers and developers, 

0 spokesmen for lines of research the institute wants to encourage ^ and 
0 at least one or two "elder sta-^esmen" of the paradigm. 
All o£ the nominees must be very near the top quality in their specialty. 
The mix must include some who are known to hold countervailing views. Im- 
portantly, all must be conversant with a large part of the research paradigm 
under review. The ability of panelists to function harmoniously and support- 
ively in a group session must be predetermined through observation in other 
panel sessions. 

FORMAT 

Some of the nominees are requested to submit a state-cf-the ^art paper 
which achieves two objectives. First, it summarizes the current state of the 
art in one approach to the research paradigm. Second, it states which topics 
within this approach should be supported in the future, based on their po- 
tential for practical and scientific results. These papers are circulated 
to the participants several weeks before the meeting. The authors are free 
to publish their work, but NIDR does not. Many of the participants prepare 
counter -papers, as some very strong positions against the current lines of 
research are taken by some authors. 

The formal session is planned by a steering committee consisting of 
three consultants chosen by the program director. These consultants also 
serve as co-chairmen of the formal session. The consultants work with the 
program director and his staff in selecting the panels, in drawing up the 
agenda for the formal session, and in nominating those who will submit 
state-of-the-art papers. A sample agenda from the Assessment Workshop on 
Dental Amalgam is as follows: 
Introduction 
Structure of Amalgam 

Phase Description and Influence of Composition on Micros tructure 
Influence of Alloy Parcicle Size and Shape 
Influence of Treatment of Alloy Ingot 
Properties 

Mechanical 

Dimensional 

Corrosive 
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Clinical Performance 

Methods of Evaluation 
Results of Clinical Studies 
Comparison with other Materials 
Recommendations for Future Research. 
The sub-entries in this agenda are what have been called research approaoheB. 
The major divisions of the agenda will be called research areas. 

The agenda is to have three phases: First, set down the record of 
research; second, summarize, assess, and relate to current basic research 
activities the results of clinical research on the usefulness of practical 
applications of the paradigm* Third, discuss the best directions for research 
to take in the future. 

Attendance at the session is limited to panelists and Institute staff* 
No observers are allowed. 

SCHEDULE 

Assessment Workshops begin with an introduction by the program director. 
He emphasizes the two purposes of assessment workshops, acknowledges the high 
caliber of the participants, and gives a briefing on meeting dynamics. He 
informs the participants that they will be asked to evaluate meetilg dynamics 
at the end of the formal session. The introduction lasts for about one-half 
hour. 

Four-fifths of the workshop is devoted to presentation of the state- 
of-the-art papers prepared by the selected participants. One paper is 
presented on each of the research approaches on the agenda. Each formal paper 
is followed by a discussion period lasting one-half hour to an hour, where 
counter-papers are presented, and the panelists debate informally over which 
topics should be pursued within the approach. During discussion there is a 
strict prohibition against debating the practical usefulness of the research 
approach. Debate only on scientific merit is allowed. The purpose of debate 
is to get agreement on the current state-of-the-art and the prospects for 
advance in the state-of-the-art. The issue of practical usefulness is 
discussed later in the workshop. 
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The last few state-of-the-art papers deal with the clinical aspects of 
the research paradigm. The subject of these papers Is to assess the per- 
formance of the system being researched in actual practice. An item of parti 
cular importance is whether or not performance has improved over the years. 
In the case of dental amalgam the question wa-j whether or not amalgam restora 
tlons last longer 9 and are easier to install than before research started. 
The last state-of-the-art paper in the Amalgam Workshop was a comparison of 
dental amalgam with other restoration materials. 

The last fifth of the meeting starts by having one of the three consul- 
tants on the steering committee present a paper that summarizes the direc- 
tions for future research that seem to be the sense of the meeting. A 
lengthy discussion period follows where the importance of research to prac- 
tice is discussed. 

OUTPUT AND DESIRED EFFECTS 

The tangible output of the Assessment Workshop is a synopsis of the 
meeting prepared by the three-man steering committee and the program director 
It is published in a top-quality dental journal. The principal requirement 
of this review is that it be short and written so that more than the insiders 
can understand it. 

An Intangible output of the Assessment Workshop, which is perhaps the 
most effective, is the reorientation and general overview provided to the 
key researchers in the field. 

One by-product of the workshop is that NIDR obtains materials useful in 
preparing budget justifications. Another by-product, although it was not 
mentioned by NIDR, is that NIDR's efforts to recruit new disciplines into an 
area are legitimized and directed. 



